Joe Rice-Jones writes that his Asus ROG Flow Z13 tablet with a Ryzen AI Max+ 395 APU and 128 GB unified memory now runs the same local LLMs as his desktop mini PCs via Lemonade, using a model router to automatically send prompts to different sized models for everyday use and agentic coding with Crush, making it his preferred device.
- Tablet is the Kojima Productions Ludens special edition ROG Flow Z13
- Radeon 8060S iGPU has 96 GB of memory allocated in Adrenalin software
- Model tiers include Qwen3.5-2B to gpt-oss-120b MXFP4 ~59 GB on disk
- 120B model loads in 27.6 s, first token in 0.86 s, ~52 tokens/s on iGPU
- Misrouted CPU-only run dropped to 0.44 tokens/s, over 100x slower
- Router uses Qwen3.5-0.8B classifier and logs which rule fired per response
- Crush coding required raising Lemonade context window from 4,096 to 32,768 tokens
- Tablet is the Kojima Productions Ludens special edition ROG Flow Z13
- Radeon 8060S iGPU has 96 GB of memory allocated in Adrenalin software
- Model tiers include Qwen3.5-2B to gpt-oss-120b MXFP4 ~59 GB on disk
- 120B model loads in 27.6 s, first token in 0.86 s, ~52 tokens/s on iGPU
- Misrouted CPU-only run dropped to 0.44 tokens/s, over 100x slower
- Router uses Qwen3.5-0.8B classifier and logs which rule fired per response
- Crush coding required raising Lemonade context window from 4,096 to 32,768 tokens
Qwen Team writes that Qwen3.8-27B is presented as the most capable generation in the Qwen open-model family so far, built on the Qwen3.5 foundation with substantial gains across coding, professional work, research and long-horizon agentic tasks. The model is a 27B-parameter dense causal language model with a vision encoder, native 262,144-token context extensible to 1,000,000 tokens, flexible thinking control with reasoning_effort and preserve_thinking, and Multi-Token Prediction for faster inference. The Hugging Face page hosts Unsloth's GGUF quantizations and provides install and run instructions for llama.cpp, Ollama, Unsloth Studio, LM Studio and other local apps.
Shuai Guo writes about implementing structured output with local LLMs to ensure responses are easily consumable by software applications. By using Pydantic models and the Ollama runtime, developers can constrain model generation to follow specific schemas, transforming unstructured text into predictable Python objects. The author demonstrates a smart-home use case where data is sanitized for downstream processing while maintaining privacy via local execution.
- Validating structure does not guarantee content accuracy or logical correctness.
- Complex tasks are better handled through task decomposition (staged approaches).
- Local LLM deployment helps protect sensitive household or personal information.
- Validating structure does not guarantee content accuracy or logical correctness.
- Complex tasks are better handled through task decomposition (staged approaches).
- Local LLM deployment helps protect sensitive household or personal information.
Nolen Jonker writes about how he transitioned from using Claude Code to running local LLMs equipped with filesystem access via the Model Context Protocol (MCP) for administrative tasks. He notes that while Claude is superior for complex coding, a local model can handle folder organization and file management without the privacy concerns of sending sensitive data to cloud servers or being subject to rate limits.
- The filesystem MCP server is an official Anthropic-maintained Node.js package available via npm.
- Setting up the tool in LM Studio requires only a single edit to a `mcp.json` file.
- Running Qwen 3.5 9B on 8GB of VRAM provides sufficient capability for reliable multi-step sequences and tool calling.
- The filesystem MCP server is an official Anthropic-maintained Node.js package available via npm.
- Setting up the tool in LM Studio requires only a single edit to a `mcp.json` file.
- Running Qwen 3.5 9B on 8GB of VRAM provides sufficient capability for reliable multi-step sequences and tool calling.
>"I Measured Every Watt on Apple Silicon Five models, sustained generation, real wall-socket energy at $0.31/kWh — and the surprise the RTX-3090 numbers predicted, only bigger."
Justin Stewart writes about how the energy cost of running local Large Language Models (LLMs) on Apple Silicon depends more on throughput than parameter count. Using an M3 Ultra Mac Studio, he demonstrates that large Mixture-of-Experts (MoE) models can be significantly cheaper to operate per token than smaller dense models because they only activate a fraction of their parameters during generation. Ultimately, the study reveals that efficiency is driven by how much data must be moved from memory for every token produced.
* The measurements were calibrated against actual wall power using a Shelly Plug US Gen4 meter.
* A custom tool called TokenWatt was used to measure marginal energy consumption via Apple’s IOReport interface.
* In real-world "lumpy" traffic scenarios, the cost of dense models compared to MoE models actually widens even further.
Justin Stewart writes about how the energy cost of running local Large Language Models (LLMs) on Apple Silicon depends more on throughput than parameter count. Using an M3 Ultra Mac Studio, he demonstrates that large Mixture-of-Experts (MoE) models can be significantly cheaper to operate per token than smaller dense models because they only activate a fraction of their parameters during generation. Ultimately, the study reveals that efficiency is driven by how much data must be moved from memory for every token produced.
* The measurements were calibrated against actual wall power using a Shelly Plug US Gen4 meter.
* A custom tool called TokenWatt was used to measure marginal energy consumption via Apple’s IOReport interface.
* In real-world "lumpy" traffic scenarios, the cost of dense models compared to MoE models actually widens even further.
This guide outlines the most effective approach for running large language models locally on hardware with 24GB of VRAM. It advises moving away from squeezing extremely large parameter models toward using high-performance 20B to 35B class models that allow room for context and fast processing speeds. The article explains how memory is allocated across model weights, KV cache, and runtime overhead while recommending specific top performers:
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
* Qwen3.6-27B for agentic coding
* Qwen3.6-35B-A3B MoE for speed in general conversation
* Gemma 4 26B for multimodal and multilingual support
* Mistral Small 3.2 24B as a low-latency assistant
* gpt-oss-20b for structured reasoning tasks
* DeepSeek-R1-Distill-Qwen-32B for deep logical reasoning through chain of thought
The author examines the trade-offs between running large language models via Ollama versus llama.cpp's WebUI. While Ollama is celebrated for its user-friendly interface and effortless model management, it introduces a small performance overhead by acting as an abstraction layer over the inference engine. In contrast, llama.cpp provides faster generation speeds and highly granular control over hardware offloading and sampling parameters. Although llama.cpp requires more manual effort to launch and switch models, it is preferred for users seeking maximum performance and technical customization.
* Performance comparison of local LLM inference
* Comparison of user experience versus technical control
* Evaluation of abstraction overhead in AI tools
* Performance comparison of local LLM inference
* Comparison of user experience versus technical control
* Evaluation of abstraction overhead in AI tools
An experiment exploring whether a local large language model can manage a home server without strict guardrails. By using the Pi agent harness and the Qwen3.6-35B-A3B model, the author successfully enabled an LLM to control a Proxmox VE node. The setup demonstrated impressive capabilities in managing LXC containers, checking system metrics, creating snapshots, and provisioning new virtual machines. Despite these successes, the system encountered difficulties with guest terminal command execution, persistent configuration management, and multi-node scaling.
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
- Implementation of the Pi agent harness with local models
- Autonomous creation of custom Proxmox extensions by the LLM
- Successful management of LXC containers and VM provisioning
- Challenges with terminal command execution and credential persistence
Simon Willison reviews Ornith-1.0, a new series of open-weights models from DeepReinforce designed for agentic coding. Built on Gemma 4 and Qwen 3.5, these models range from 9B to 397B parameters and demonstrate high proficiency in coding benchmarks and tool-use tasks.
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
- Available in 9B Dense, 31B Dense, 35B MoE, and 397B MoE variants
- Capable of navigating codebases and executing complex tool calls
- Built on Apache 2.0 licensed base models
From installing Ollama to launching OpenCode with a local model, step by step.
This guide explains how to build a fully local AI coding agent by integrating Ollama, Google's Gemma 4, and OpenCode to ensure data privacy, cost control, and local experimentation. The workflow involves installing Ollama to serve the edge-optimized `gemma4:e4b` model, setting up OpenCode as the agent runtime via npm, and linking them through a local API configuration with a 128K context window. This setup provides a capable, offline development assistant that handles code generation, repository analysis, and file operations without transmitting any data to the cloud.
* **Core Stack:** Uses Ollama for model serving, Gemma 4 (`gemma4:e4b`) as the local LLM, and OpenCode as an open-source agent interface.
* **Setup Steps:** Install Ollama, pull the Gemma 4 model, install OpenCode via `npm install -g opencode-ai`, and connect them by creating a Modelfile for the context window and configuring `opencode.json`.
* **Capabilities:** The local agent supports 128K context, enables coding tasks and workspace automation, and operates entirely offline for maximum security.
This guide explains how to build a fully local AI coding agent by integrating Ollama, Google's Gemma 4, and OpenCode to ensure data privacy, cost control, and local experimentation. The workflow involves installing Ollama to serve the edge-optimized `gemma4:e4b` model, setting up OpenCode as the agent runtime via npm, and linking them through a local API configuration with a 128K context window. This setup provides a capable, offline development assistant that handles code generation, repository analysis, and file operations without transmitting any data to the cloud.
* **Core Stack:** Uses Ollama for model serving, Gemma 4 (`gemma4:e4b`) as the local LLM, and OpenCode as an open-source agent interface.
* **Setup Steps:** Install Ollama, pull the Gemma 4 model, install OpenCode via `npm install -g opencode-ai`, and connect them by creating a Modelfile for the context window and configuring `opencode.json`.
* **Capabilities:** The local agent supports 128K context, enables coding tasks and workspace automation, and operates entirely offline for maximum security.
